Machine learning editorial pairing with predictive queries influencing user behavior

ABSTRACT

Disclosed herein is an application enabling users to interface dynamically with live sports. Algorithmic generation of user-tailored editorial content that influences sports-bettor behavior, to increase engagement and manage risk. By showing bettors editorial content that could influence the side of the bet they might wish to take (for example, by pointing out that a certain player is playing particularly well or poorly), a sportsbook can influence the ratio of bets (or better, dollars wagered) on either side of a given event. On a more individual level, by showing users content that increases their engagement—such as content about a favorite player or local team, for example—the book can increase a user&#39;s propensity to bet in general, with a good sense for which types of bet the user is likely to favor.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 16/719,825, filed Dec. 18, 2019, which claims the benefit ofthe following provisional applications: U.S. Provisional PatentApplication Ser. No. 62/914,963, filed Oct. 14, 2019; U.S. ProvisionalPatent Application Ser. No. 62/885,512, filed Aug. 12, 2019; and U.S.Provisional Patent Application Ser. No. 62/830,157, filed Apr. 5, 2019.The aforementioned applications are incorporated herein by reference intheir entirety.

TECHNICAL FIELD

The disclosure relates to the continuous presentation of procedurallygenerated queries to users paired with editorial commentary.

BACKGROUND

Sportsbook betting is slow and based on pre-established queriesoriginating before relevant athletic contests begin. For example, duringthe National Football League (“NFL”) Super Bowl, often a number ofproposition bets (“prop bets”) are created long before the game beginsand are unchanging while the game is in progress. There is a need toimprove upon schemes in order to provide a more dynamic experience wherebetting queries are generated during games and based on the action thatis happening as the game progresses.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart illustrating an application flow.

FIG. 2 is a block diagram illustrating a machine learning systemexecuting a query ranking operation.

FIG. 3 is a screen shot of a query selection screen.

FIG. 4 is a flowchart illustrating insertion of editorial comments withqueries.

FIG. 5 is a block diagram of a user-tailored editorial content pairingplatform.

FIGS. 6A and B are illustrations of a user interface includingstatistical editorial comments.

FIGS. 7A and B are illustrations of a user interface including graphiceditorial comments.

FIG. 8 is an illustration of a user interface including non-athleticeditorial comment.

FIG. 9 is a flowchart illustrating operation of a machine learning modelthat pairs queries with editorial content.

FIG. 10 is a high-level block diagram showing an example of a processingdevice that can represent a system to run any of the methods/algorithmsdescribed above.

DETAILED DESCRIPTION

Disclosed herein is an application enabling users to interfacedynamically with live sports. During one or more athletic contests(multiple may occur simultaneously), the application receives input froma feed of the athletic contest (e.g., via a media feed or a gamblingfeed) describing events occurring in the given contest. From the contestfeed input, the application generates a number of queries (e.g., WillStephen Curry make a 3-point shot in the next two minutes? Will TomBrady get sacked during his next drive? Will player X score points inthe next Y minutes?) concerning the relevant contest(s). The queries aregenerated during the contests, preferably each second (˜50 per second,per contest). A machine learning model operating on the backend of theapplication ranks the queries according to a set of criteria. Queriesmay then be delivered to users in ranked order.

In some embodiments, the queries operate on a binary basis (e.g.,yes/no, this/that, or over/under). When presented with a query, theusers may swipe on their user interface to indicate an answer to thequery. In some embodiments, the user may answer “yes,” or indicatedisinterest in the query only. Queries pend on the user interface for abrief period (e.g., 10-30 seconds, or 15 seconds). Queries preferablyoperate in the time scale of minutes of game time. The queries concerngame actions that may occur next, or soon in the relevant contest(s).When one query is resolved, either through user engagement or byexpiring due to time lapse, a new query is presented to the user. Userswho engage positively with queries that correctly predict mid-contestoutcomes are awarded points, money or some other beneficial reward.

In some embodiments, each query is presented with some additionaleditorial content. The editorial content is designed to influence usersto choose a target answer to the associated query. The editorial contentmay include details such as statistics about the relevant entity. Forexample, if the query is an over/under type question corresponding toStephen Curry making 3-point shots in the next 3 minutes, then theeditorial may have statistics about Stephen Curry's tendency to shootand make 3-point shots. Where the target choice is that Curry achievesthe over, the editorial may be “Stephen Curry is the leading 3-pointshooter on the Warriors.” Where the target choice is instead the under,then the editorial may be “Stephen Curry tends to sub out at X time,”where the X time is less than 3 minutes away. Other examples ofeditorial may include graphics (e.g., an ice cube or a flame emoji), orfactoids that have nothing to do with the athletic contest.

The editorial is paired with queries based on a machine learning modelthat indicates either a user base or a user specific tendency to choosethe query engagement option that is beneficial to the platform. Themodel is trained with pairings of queries and editorial, along with thechoices picked by various users for the given pairing. Over multipleiterations, the model becomes more accurate at guessing how a giveneditorial will influence the choices of the user when presented with agiven query.

FIG. 1 is a flowchart illustrating an application flow. In step 102, abackend application server monitors one or more athletic contests (e.g.,NFL games, National Basketball Association (NBA) games, Major LeagueBaseball (MLB) games, National Hockey League (NHL) games, Major LeagueSoccer (MLS) games, games in college leagues, games occurring innon-American leagues, international competitions, video games ande-sports, etc.) via an input feed(s) of all relevant athletic contests.The input feed describes events occurring in each contest (e.g., aplay-by-play, stats crediting, game actions, notice of commercialbreaks, etc.). When games occur simultaneously, within the same leagueor across leagues/sports, multiple feeds are monitored.

In step 104, the backend application server generates queries based onthe input feed(s). In some embodiments the queries are yes/no format. Insome embodiments, the queries are multiple choice. In some embodiments,the queries include short answer fields. The queries are procedurallygenerated based on conditions within each sports game being monitoredvia the input feed(s). Procedural generation of questions is performedby generating all permutations of a number of query formats that includea predetermined number of variables. In some embodiments, the system isconfigured to heuristically reduce the number of permutations (N) byfiltering based on the game feed (e.g., values for variables that wouldrefer to teams/players not playing are not generated). In someembodiments, queries are not actively generated and instead apre-generated list of possible queries is repeatedly subjected to step106 described below. Pre-generating queries removes computation time forstep 104, and instead burdens the processing expense of step 106 below.

Queries may include permutations of, “Will player/team X perform gameaction Y, within Z time?” for all active games, each second of the game.The generated queries shift the variables to different possible values.In some embodiments, queries are generated for every possiblepermutation of variables. In the (X, Y, Z) example, the X value is foran actor (e.g., player(s), team(s), position(s) held by player(s)), thevalue of Y is for an action (e.g., score points, assist, takeaway, etc.. . . ), and the value for Z is a time component (e.g., absolute clocktime, relative game clock time, or variable, but identifiable temporalsegments like plays or ball possessions). In some cases, these valuescan have an associated magnitude. If the action component is points, themagnitude may be the number of points referred to in the query. If thetime component is seconds, the magnitude may be the number of seconds.

The game actions are sport specific, for example, queries would notreference a baseball player when asking if the player would score atouchdown. Accordingly, the possible variables include assigned metadatathat indicates which sport/team/position the player or action isassociated with. In some embodiments, queries are only generated forfeasible circumstances. For example, no query is generated for playerswhom are not actively present in the game (defensive players while theoffense is on the field, bench players whom just left thecourt/rink/field), or players who are injured and out.

The sample queries provided above serve as illustrative examples and arenot limiting on the query style/type. Queries may refer to multipleplayers, multiple teams or multiple actions. Each query includes a setof metadata. The metadata is based on any combination of the querystyle/type, the variables used within the specific query and the gamestate when the query is generated.

In step 106, a machine learning model operating on the backendapplication server ranks the queries according to a set of criteria. Insome embodiments, the queries are ranked both on an application-widelevel and again on a personal level. When individual users make choicesto engage with a particular style of query (e.g., a user whom regularlyengages with queries pertaining to scoring touchdowns), that engagementincreases the rank of future queries with a similar style. Similarly,the users may apply specific filters to the queries, such as only tosports they are interested in (e.g., a user who has no interest inbaseball can apply a filter to de-rank all baseball-related queries).

Queries are ranked on an application-wide level. Ranking of queries isbased on a number of factors directed at targeting user “excitement.”Queries that pertain to more popular players or more exciting actionsare given a higher rank (e.g., asking whether the star player will scorepoints is ranked higher than asking whether a less popular playerassists on the score). In some embodiments queries are additionallyranked based on application-wide user engagement. The manner in whichapplication-wide user engagement affects ranking of queries is based onthe user reward structure of the application. In a betting scenario, ifone query becomes over leveraged by the user base, the application mayde-rank that query in order to reduce overall risk on that query, orup-rank the opposite side of the over-leveraged query in order tobalance risk. If a given query is trending positively in engagement butthe application is not overleveraged, the model up-ranks the query inorder to place that query in front of more users. In a casual gameplayscenario where leverage of the house on a particular query ismeaningless, positive trends in user-base engagement serve to up-rankthe query.

In step 108, queries are then delivered to users, one at a time, inranked order. The queries appear on a user device running a clientinstance of the application. In step 110, the instance of theapplication waits for engagement by a user. In step 112, where the userdoes not engage within a threshold period of time, the current querytimes out and is replaced with a new query in step 114. In step 116,where the user engages with a query, the user's stake/association withthe query is logged and the logged query is replaced with a new query instep 114.

In step 118, logged queries are compared with the input feed for as longas they are active. Where the user correctly predicts game actions (viaa logged query), the user is awarded the associated reward for beingcorrect. In some embodiments, the reward is greater if the user wascorrect multiple times in a row. In step 120, the application determinesif there are sports games still in session. If there are games insession, the process repeats.

FIG. 2 is a block diagram illustrating a machine learning system 200executing a query ranking operation. Query ranking is similar togenerating a search rank in a search engine, where the search engineonly operates with one question: “What is the best query in sports rightnow?” The machine learning system operates on a backend applicationserver 202. The machine learning system may be constructed usingmultiple models. Examples of potential model architectures includehidden Markov models, convolutional neural networks, and deep learningnetworks. Each of the models may be supervised, unsupervised orsemi-supervised. In some cases, pre-programmed artificial intelligenceand heuristics operate to come to arrive at query ranks.

The backend application server 202 communicates directly with userdevices 204 through client application software instances executing onthe user devices 204. The client application software provides a graphicuser interface for the individual users. Through the graphic userinterface, users engage with queries supplied by the backend applicationserver 202.

The backend application server 202 further communicates with theInternet 206 via web crawlers 208 that return data about websites to thebackend application server 202 for processing. The backend server 202further communicates with an input feed 210 that delivers up to themoment information about active athletic contests. In some embodimentsmultiple input feeds 210 originating from differing sources communicatewith the backend server 202. The input feeds 210 corresponding todifferent games or different leagues may originate from differentsources. In some embodiments the input feed 210 is a media feed providedby the relevant league to media outlets for the purpose of broadcastingcontent related to the athletic contest. In some embodiments the inputfeed 210 is a gaming or gambling feed used to provide information togambling outlets concerning the outcome of athletic contests.

Factors that the machine learning models uses to evaluate that questioninclude: game affinity, player statistical affinity, participantaffinity (specific to that player-game relationship), end user affinity(personalization based on user's preference), operational affinity(features based on the house's goals), other user affinity or userbaseaffinity (a given user's behavior is similar to other users who may becategorized into archetypes or sub-archetypes either expressly or viatraining of a machine learning model), and competition affinity(specific to a user-competition relationship). Metadata associated witheach query is used to evaluate as compared to the factors.

Game Affinity:

Game affinity refers to how much the particular game matters in sports.Users want to engage with queries related to games that are important. Aplayoff game, or a game against a rival, has higher priority than aregular season game. A game with a lopsided score, a game that iscurrently in intermission (halftime, commercial break, etc.), or a gamethat hasn't quite started yet are inherently less interesting and arede-ranked. Games that have current action are up-ranked. Games that havecloser scores between teams are also up-ranked.

In some embodiments, game affinity factors operate on absolute binarydata (e.g., intermission or not). In some embodiments, game affinityfactors operate on a comparative basis (e.g., there is an indirectcorrelation between the gap in team score and the value the rankingengine assigns the game). In some embodiments, game affinity factorsoperate on a pre-determined value (e.g., a regular season game is worthsame first value to the ranking engine, whereas a playoff game is worthsome second, higher value to the ranking engine). The pre-determinedvalues may be applied as multipliers or additives.

Player Statistical Affinity:

Player statistical affinity refers to the star power of a given playerthat a query is about. Each player across sports has an individual valueas compared to players at large, players in the same position, playersin the same age group, players from the same school, players from thesame team, or players from another objectively defined group. Theindividual score is based on in-game statistics, media buzz, and userengagement. In some embodiments, the individual value is applied on aper-action basis (e.g., a player may have a higher value assigned tohim/her when performing assists than scoring because he/she is known forthat action).

In-game statistics may be objectively compared to one another on aone-to-one basis with matching statistic types. In some leagues/sports,there is an overall type statistic that is provided more weight thanother statistics (e.g., hockey awards points for both goals and assistsand football has a quarterback rating formula that is an amalgamation ofmultiple statistics). Some statistics are weighted as more impactful onthe player's individual score in the query ranking system.

In some embodiments, players are assigned a Z-score that is used toevaluate the player rank. The Z-score operates cross-statistic andcross-sport. The Z-score is standardizes statistics of players relativeto their peers within the league and then normalized between 0 and 1. Tofind the Z-score of a player, find the difference between a value forthe player at a given statistic and the mean for that statistic at thatplayer position, and divide the difference by the standard deviation forthat statistic at that player position. Each Z-score is then scaledbetween 1 and 0 (e.g., x_new=(x−x_min)/(x_max−x_min)).

For example, if Player A produces rebounds at a rate of 0.30 per minuteand Player B produces points at a rate of 1 per minute, then the Z-scoretransform turns these each into 0.95 and 0.91 respectively. The unitsfor the Z-score are “statistical affinity” and may be applied evenlyacross all athletes and all sports. The ranking system is enabled toclaim that Player A has a higher statistical affinity than Player B. TheZ-score is applied inter-sport where the ranking system can comparePlayer C's yards per carry as a running back to both Player A and PlayerB's basketball statistics via pure statistical affinity.

Media buzz is identified on an objective basis. In some embodiments, aweb crawler is used to identify the number of unique pages on theinternet that reference the given player (or team). In some embodiments,audio recognition is applied to sports television (e.g., ESPN'sSportsCenter) and instances of recitations of each player's name aretallied. Appearances on some websites or television programs areweighted more highly than others (e.g., reference on ESPN's homepage isweighted more highly than a forum associated with an unaffiliatedblogger). The value of a given page over another can be configured on apredetermined basis or on an objective basis that evaluated how recentlythe page was authored/updated as well as how many times that page isreferenced by other pages.

User engagement is identified as an ongoing statistic affected by usersengaging with queries generated by the system. When a user engages witha query associated with a particular player (or team), that playersengagement score goes up. In some embodiments, the engagement scoreapplies to an overall query rank in a logarithmic or asymptotic manner.Because engagement drives further engagement, effecting a searchrank/query rank score with a direct correlation or better (e.g.,linearly, exponentially, etc.) causes a snowballing effect that candominate the system. Statistics that are generated in-system and affectsearch rank within the system can lead to having too great an effect onthe system.

User engagement may be further used to influence query type (as opposedto player choice). For example, where users of the system tend to engagewith queries regarding a particular game action (e.g., whether a kickingteam in football will recover an on-side kick), the system up-ranksqueries for that action. Query type may also refer to whether thequeries are yes/no, multiple choice or short answer. Query type mayfurther refer to a risk/reward profile of the query. Queries may includea cost to engage with and reward some value of points based on aperceived likelihood of correctly predicting game events (e.g., scoringa safety in football is a relatively low occurring event, and thus,betting in favor of a safety has a higher risk). The opposite effect torank is applied for when users allow a given query to time out andexpire or send input that they are disinterested in that query. Queriesthat the user base does not engage with are down-ranked.

In some embodiments, queries regarding players on a hot streak for agiven statistic are up-ranked. After the ranking system receivesnotifications from the input feed (indicating game status) that a givenplayer has exceeded a threshold statistic accumulation in a thresholdtime, the ranking system determines that player is on a “hot streak” andup-ranks queries that ask users whether the hot streak will continue. Insome embodiments, the existence of a hot streak modifies the magnitudefor variables within a given query.

Participant Affinity:

Participant affinity relates to the manner in which the given playerassociated with the query relates to the game he/she is playing in. Insome embodiments, the player is worth more to a ranking if the player isa starter or sees significant time playing as opposed to a bench player.Starting players are valued higher than substitutes, and substitutes arevalued higher than substitutes of substitutes (e.g., third string).

When the player is injured (questionable, doubtful, out, etc.) theplayer is de-ranked as less exciting to the particular contest. When theplayer is actively on the field/court/rink/ring/octagon/sports playingarea, that player is up-ranked. Similarly, when the player's team haspossession of a relevant game element (e.g., a ball) the player isup-ranked as well as that player is more likely the subject of anexciting query. In the case where the player is a defensive player, nothaving possession of the relevant game element up-ranks the playerinstead (e.g., queries relating to defensive players like blocks andsteals are more exciting when their team does not have possession).

End User Affinity (Personalization Based on User's Preference):

End user affinity refers to user specific effects on search/query rankthat the user has an active role in influencing. The user can indicateto the application which sports he/she is most interested in and whichplayers and teams he/she is interested in. User preferences may beapplied as either a filter (e.g., removing all queries that do not matchuser preference) or a bias (e.g., providing additional weight to therank of queries that match the user preference over other queries). Userfilters or biases may be configurable on a per sport or a per teambasis. For example, a given user may be interested in all teams from theNFL, but only a single team in the NBA. While interested in all NFLteams, he/she may bias a particular team or set of players.

In some embodiments, end user affinity is further affected by thephysical location of the user. A user located in a city where a givenathletic contest is occurring up-ranks queries relating to that contestand players therein. Similarly to user engagement en masse, pastengagement (or lack thereof) by a single user affects the query rank offuture generated queries for that user. A given user may influencefuture system query ranking (for himself/herself) based on the relevantquery type, players, teams or sports they engage with queries for.Through user actions, a user may generate filters or biases over timewithout expressly creating those filters or biases. Unlike actions enmasse, there is no snowball problem when applying a direct correlation(or a more effective correlation) between an individual's engagementhistory to query ranking. Where a specific user prefers a certain methodof engagement, enforcing that engagement pattern does not affect theengagement pattern of other users.

Operational Affinity:

Operational affinity relates to goals held by operators of theapplication server. For example, where “the house” is operating abetting scenario via delivered queries, it behooves the house to modifythe factors by which queries are ranked. For example, in someembodiments, the house affects the ranking algorithm tominimize/maximize exposure on a given query. Should the user base win ona given query, the house does not want too great a percentage of theuser base to be engaged in that query. The effect of the house'sinterest in a given query operates on a parabolic curve relating to userengagement. To a point, user engagement is encouraged. Once a certainposition is reached (depending on size of the user base, and thepercentage of the user base engaging with an opposite position oropposing position of a similar query), user engagement is discouraged.Conversely, a query that is structured in the opposite or is related andopposing becomes encouraged as the positive query is engaged with.Opposite may refer to a pair of queries that are the same with theexception of the word “not”, “won't” or other negating terms. In somecases, an opposite query takes a position that is a mutually exclusivealternate of a position held by another query. The degree to which agiven query is encouraged or discouraged in the ranking algorithmoperates on the slope of the parabolic curve.

A query that is related and opposing is one that is not necessarilymutually exclusive but is unlikely in view of the success of another.For example, if a first query poses an over/under on the total score ofa given game, a related query is the over/under on the number of pointsa given star player scores. That is if the star player does poorly, itis unlikely that the over on the total game points will be reached.Thus, the under on the total score of a given star player is related andopposing the over of the total score of the entire game.

In some embodiments, the ranking algorithm may be configured to pushlong shot queries (e.g., a query that is in favor of a given footballteam scoring 21 points in two minutes of game clock), or promotionalincentives (e.g., application operator interest in a particular game).Promotional queries are generated through direct intervention byapplication administrators in the ranking algorithm.

Competition Affinity:

Competition affinity refers a given user's performance in correctlypredicting game events as compared to other users. In some embodiments,users who engage with queries that result in correct predictions moreoften are provided additional rewards for “hot streaks” (with respect tothe user, as opposed to athlete actions). In a given day, or otheroperative unit of time, system users are matched up on a leaderboard.Queries each have a risk and reward proposition associated with them.Users trying to catch the leader may have queries associated with ahigher risk/reward profile up-ranked for them. Users in the lead mayhave queries associated with an even risk/reward profile up-ranked forthem in order to maintain their position. As time remaining on a givenleaderboard competition gets close to ending, queries that enablegreater movement on the leaderboard are up-ranked. High risk/high-rewardqueries tend to lead to greater leaderboard movement and are accordinglyup-ranked.

In some embodiments, leaderboards are configured based on physicallocation of users, where users whom are in the same geographic regionare grouped together. Users within the same leaderboard up-rank queriesthat serve competitive balance between those users. Where users of agiven leaderboard tend to engage with a particular query, that query isdynamically up-ranked so that other users associated with thatleaderboard are presented that query.

Ranking Formula:

References to up-ranking and down-ranking in response to a given factormay be implemented as additive to a total rank score, or as a scalar tothe total rank score or a portion of the rank score. The magnitude of anadditive or scaler depends on weighting choices of a particularembodiment. Shifts in whether a given factor causes an up-rank or adown-rank may be influenced by absolute or dynamic thresholds regardingvalues held by those factors.

Magnitude Adjustment Feedback Loop:

The way users engage with queries further informs the magnitudes appliedto future queries. A given query is “will player X, score Y points in Ztime” and that query is offered with certain odds. The relevantmagnitudes are the values of Y, Z, and the odds. Based on the number ofusers whom engage with a query at a certain set of magnitudes, thesystem may adjust the magnitudes to reduce or to increase engagement offuture queries. Based on the given variable, and whether the goal is toincrease or decrease engagement on a given query type, magnitudes arealtered. For example, requiring more points before a predictive query issatisfied reduces engagement because the likelihood of predictivesuccess is lower. Conversely, increasing the amount of time allotted forthe prediction to come true increases engagement because the likelihoodof predictive success is greater.

In some embodiments, the queries: “will player X score at least 5 pointsin 2 minutes?” and “will player X score at least 10 points in 2minutes?” will rank the same in the ranking system. However, one wouldexpect the engagement of the former query to be greater because theformer query is objectively easier to satisfy. Based on targetengagement goals, the magnitudes of future queries are adjusted. Becauseof the rate at which queries are generated and put into circulation infront of users, the feedback loop has a dynamic effect on the queriesgenerated and presented. Traditionally sports betting activity readjustson a per game basis. If one bet was over engaged with, magnitudeadjustments do not go into effect until additional bets are decided fora following game (at least 24 hours later). The feedback loop disclosedherein enables updates to generated queries in time measured in seconds.

In some embodiments, magnitudes are further adjusted based on inputfeeds. If a given player is on a hot streak in their game, magnitudesfor queries may be less generous towards the users engaging with thequeries. “Generosity” is determined on a variable-by-variable basisdepending on the sport and players involved in the query.

FIG. 3 is a screen shot of a query selection screen. Depicted in thescreen shot is a menu that includes all active athletic contestspresently occurring. A user may select a given contest from the menu inorder to narrow the field of queries that they receive (e.g., queriesdirected to other contests are filtered out of the query rankingprocess). At the bottom of the menu, a “Dealer's Choice” buttonindicates a willingness to accept queries relating to any activecontest. The menu indicates a current status of each contest and therelevant league.

FIG. 4 is a flowchart illustrating insertion of editorial comments withqueries. During the course of serving queries to users, the systemfurther pairs queries with editorial content. The editorial content isemployed to influence users toward a particular or target response tothe query. Target choices are determined in a number of ways. Examplesof schemes to choose target queries include balancing out platform-widerisk in a betting scheme (e.g., where users have overwhelmingly chosenone option of a query, the platform is encouraged to have users choseanother option) or encouraging users to take riskier or safer choices.Target choices improve the operational affinity of the platform in amanner that provides an advantage to “the house” and potentiallyinfluences users to make choices that are statistically against theirinterest.

In step 402, generated queries are ranked and prepared for presentationto a user in any embodiment described herein. In step 404, the platformidentifies a target choice for the queries presented on either aper-user, user-group or user-base basis. The target choice may bedecided either as a just-in-time determination for a current displayedquery, or preemptively on all ranked queries in a given set of queries.When the target choice is determined just-in-time, the target choice maybecome varied or change based on what the previous target choice was,and how the user or user base responded en masse to editorial influence.The target choice is determined via operational affinity.

In step 406, the platform pairs editorial comments with the queries. Insome embodiments, the content of the editorial comment is based on ahistory of user base query engagement that indicates that the contentencourages users to choose the target choice from among multipleengagement choices of each query. In some embodiments the content of theeditorial comment is categorized via meta tags associated with variouseditorial content. The meta tags associate procedurally generatededitorial content with any of positive or negative sentiments, actor,action, or time components, magnitudes, and a frame of reference. Amachine learning model (e.g., a hidden Markov model, a neural network,etc.) may use the meta tags for purposes of categorizing input/output.

Generation of the editorial comments is based on factual data stored ineither an internal database or external reference database. The factualdata is applied to a set of templates that are semanticallycomprehensible when variable strings based on the factual data areinserted into blank spaces.

In step 408, once paired, the editorial comments are displayed with thecorresponding query influencing users toward engagement with the targetchoice.

FIG. 5 is a block diagram of a user-tailored editorial content pairingplatform 500. The platform 500 includes a number of components includinga live game-feed(s) 502, an algorithmic query generator 504, aneditorial content generator 506, and an optimizer 508.

The live game-feed(s) 502 receive game updates of current athleticcontests which are sufficiently rich as to enable thealgorithmic/procedural creation of editorial content. The feeds 502provide stats for each player and team involved in a given athleticcontest. In some embodiments additional stats may be directly derivedfrom raw feed data (e.g., comparisons between teams/players that are notcurrently playing each other). In some embodiments, data is scraped fromthe web in real-or-near-real time (e.g., a twitter crawler that puts uprelevant tweets that have been algorithmically evaluated using sentimentanalysis or other forms of natural language processing).

The algorithmic query generator 504 automatically generates queries andassociated bets based on underlying game conditions, as well as one ormore models generally built using advanced statistical techniquesincluding, but not limited to, machine learning. The algorithmic querygenerator 504 takes as its inputs historical data, as well asinformation from live game-feeds. Various embodiments of the querygenerator are described above, herein.

The editorial content generator 506 communicates with the live game-feed502 and implements historical data to generate a plurality of editorialcontent including, but not limited to, “factoids,” “graphics.” “relevantstats.” and “trends.” For example, it might generate a headline such as“Steph Curry is HOT tonight” based on the fact that Curry is convertingshots at a rate above his career average in the ongoing contest. Whetherthat particular bit of content would be shown to a given user. is up tothe optimizer 508.

The optimizer 508 intelligently blends the live game feed 502 with theoutput of the editorial content generator 506 to create an improved userexperience. The intelligence of the optimizer 508 is based on anymachine learning and/or heuristics.

The optimizer 508 evaluates a given user's behavior, as well as theoverall risk profile of the platform/sportsbook, and sifts through theplurality of generated editorial content in order to (a) determine whattype of content, if shown to the user, is most likely to generateadditional engagement, and (b) what, if any, adjustments need be made inlight of the current and desired risk profiles of the sportsbook. In thecase of a betting feed (wherein a user is shown a subset of availablebets at a time), the optimizer 508 may also help determine which betsare shown to the user in what order in addition to what editorialcontent (if any) is paired with each.

While a live event is underway, the algorithmic query generator 504generates “raw” bets—bets that would be offered were the sportsbook toknow nothing about the person to whom the bets will be shown. Meanwhile,the editorial content generator 506 begins generating a plurality ofbits of content, which are classified into various bins (e.g., “ProWarriors” “Steph Curry +” “Steph Curry −” based on the type of sentimentviewing such a piece of content is likely to generate in the user). Theoptimizer 508 (a) determines what sorts of bets the sportsbook wouldmost like this user to make or engage with, and (b) searches theeditorial content to find the content most likely to generate thedesired user behavior. Finally, the user is shown the bet(s) andpiece(s) of content, with the user's choice(s)—to engage or not, and towhat degree—noted by the optimizer for future optimization. This cyclerepeats on a per-user basis quite frequently, in some cases more thanonce per second (this cadence, itself, might be evaluated and optimizedby the optimizer 508).

FIGS. 6A and B are illustrations of a user interface 600 includingstatistical editorial comments 602. Two separate statistical editorialcomments 602A, B are displayed with the same query 604 (whether thepresented player, Clint Capela, will achieve at least one rebound,expressed as an over/under prompt, in the next 180 seconds of gametime).

A first statistical editorial comment 602A reads, “Capela is theRockets' leading rebounder this season” and presents a player statistic,rebounds as compared to others, that leads a user to believe that thesubject entity, Capela, will accomplish the “over” prompt.

Conversely, a second statistical editorial comment 602B reads, “Capelausually subs out around the 10:30 mark of the 3rd quarter.” The secondeditorial comment 602B is based on current game status 606, whichindicates that the athletic contest is currently at 10:54 in the thirdquarter. The second editorial comment 502B implies that the subjectentity (again, Capela) is likely to leave the game in approximately 24seconds and thus, may not be around for all of the time period that thequery corresponds to. Thus, the second editorial comment 602B presents aplayer statistic, specifically a gameplay pattern, that leads a user tobelieve that the subject entity, Capela, will come in “under” theproposed number of rebounds.

Each example demonstrates how, in the same athletic contestcircumstances 506, and with the same query 604, a different editorialcomment 602A, B is applied that leads users toward a target conclusion608 or engagement choice of the query 604 that is presented to them. Theeditorial comment applied in each case relates directly to the subjectquery 604. Specifically, the first editorial comment 602A directlyrefers to the action referred to in the query 504, rebounds, and theextent to which the acting player, Capela, performs that action. Thesecond editorial comment 602B refers to the time component of the query604—that is that the acting player may not be playing for the full timeperiod allotted for the query 604.

In some embodiments, the statistical editorial comment 602 correspondsdirectly to any of the actor, action or time components. The statistics,while true, lead users to different conclusions. The editorial commentsare determined based on a machine learning model that is trained on userbase response to the comments. If the user base is more readily directedto a target choice by editorial comments that refer directly to theaction in the query, then action based editorial comments will beemployed more frequently.

The system procedurally generates statistical editorial comments 602 viaany heuristics, templates, and sematic models that convey tone. Thecomments 602 are specific to the sport the subject query 604 relates to.FIG. 6 illustrates a query connected to NBA basketball and accordingly,includes references that are sport specific (e.g., subbing out,performing a rebound, etc.). In other sports, different templates areused that are based on the game status, and specific rules of thesubject sport/game being played (e.g., in competitive e-sports, such asLeague of Legends, the statistics might be based on creep score, or akills/deaths/assists ratio).

In some embodiments, the editorial comments are generated during theathletic contest (e.g., those based on how the current game has gone)and others are generated pre-game (e.g., those that pertain to season orcareer averages). As the system generates each editorial comment, theyare flagged with numerous metatags. Examples of meta tags include thetone of the comment (e.g., positive or negative); whether the commentimpacts the player, the action, or the time period (or some combinationthereof); whether the action is related to other types of actions (e.g.,assists and scoring plays are related); a frame of reference tag (e.g.,indicating that a player averages 30 points a game is positive whenasking whether a player whom has currently scored 10 will score more,and negative when then have scored 10 and the query asks whether therewill be yet 10 more); or how convincing the given statistic is (e.g., amagnitude component—saying a player averages 30 points is inherentlymore convincing that they will score a lot of points than stating thatthey average at least 10 points). Meta tags may be used to categorizetraining data into the machine learning model or enable heuristics tosort and pair queries to comments more efficiently.

In some embodiments, the editorial comments are further determined basedon individual user behavior in response to past query/editorialcombinations. For example, particular users may act as a contrarian.Specifically, contrarian users will choose an engagement choice otherthan the one they believe the system is directing them toward. Thus, inorder to get the contrarian to select the target engagement choice, thestatistical editorial comment should appear to direct the user toward analternate engagement choice.

FIGS. 7A and B are illustrations of a user interface 700 includinggraphic editorial comments 702. Two separate graphical editorialcomments 702A, B are displayed with the same query 704 (whether thepresented player, Russel Westbrook, will score 3 or more points,expressed as an over/under prompt, in the next 180 seconds of gametime). Much like the statistical editorials, graphics can perform asimilar function. Graphical editorial comments are selected from apremade list. Examples include the “fire” icon 702A and the “ice” icon702B, each respectively indicating that a player/team is hot or cold.Other examples that convey a similar meaning may include a graphic ofthree buckets on fire or a toilet.

In some embodiments, the graphic editorial comments 702 are employedwhen the context of the athletic contest make very clear what thegraphic implies. Among basketball fans, a player who is “on fire” is onewho has made their last three shots. Thus, the context of the gamematching the fan-meta meaning of the graphic causes users to understandand be influenced in engagement.

In other embodiments, the graphic editorial comments 702 have nofan-meta meaning and may have no inherent connection to the athleticcontest at all (e.g., a graphic of a sombrero, a graphic of a prism, ora graphic of a banana). For graphic editorial 702 with no fan-metameaning, the system runs experiments to train the machine learningmodel. Initially, the experiments arbitrarily employ the graphics withno fan-meta meaning and observe the results. Eventually, if the userbase, or even individual users. demonstrate a query engagementtrend/sentiment when shown a particular graphic (e.g., if users whom areshown the banana graphic tend to take the over, or a riskier option),the machine learning model will employ that graphic to influence userstoward a target choice.

FIG. 8 is an illustration of a user interface 800 including non-athleticeditorial comment 802. Similar to the graphic editorial that does nothave a fan-meta understanding, textual editorial that is not athletic ornot directly connected to the athletic contest may be employed initiallyvia experiments and subsequently in a targeted fashion. Pictured in theFigure is the non-athletic editorial comment 802 that “Lyles' favoritefood is borscht!” This non-athletic editorial is paired with anover/under query about rebounds 804. There is no rational connectionbetween liking borscht and making a rebound in 3 minutes of game time.

After conducting experiments and identifying a tendency of the user baseor a particular user to respond to apparently random factoids orspecifically borscht related content, the machine learning model beginsto pair the non-athletic editorial comment 802 with queries to influenceusers toward a target choice.

FIG. 9 is a flowchart illustrating operation of a machine learning modelthat pairs queries with editorial content. The machine learning model istrained via initial experiments and ongoing platform use. The pairingmachine learning model reviews a number of various factors and outputs apairing of an editorial content, a query and a user.

In step 902, the machine learning model performs a series of experimentsbased on a set of queries delivered to individual users. In someembodiments, the experiments arbitrarily pair editorial content withqueries according to heuristically identified relationships/nexusbetween the editorial and the queries (e.g., such as through matchingmeta tags between the queries and editorial). If the experiment wasconducted using a hidden Markov model, the query, the user and theeditorial are all states whereas the user response or engagement choiceis the emission. Once the user selects a given engagement choice, theexperiment is logged. The experiments inform results for both theindividual user as well as how users react overall.

Over the course of many iterations of the experiment, the model developsassociations between various editorial content and queries in a mannerthat develops computational predictive confidence toward how a specificuser, or a generic user will engage with a given query based on displayof a specific editorial content.

In some embodiments, the states are even more granular and take intoaccount where the user lives, what teams the user favors, or how manytimes similar editorial content had previously been shown to that user.It may be the case that after conducting a sufficient number of theexperiments that a trend appears that New England Patriots fans arefound to be contrarians and tend to pick the engagement choice oppositeto what the platform appears to be directing the user to.

In step 904, after experiments have been conducted and a model istrained, the model pairs editorial content with queries, but this timebasing the pairing on a target engagement option.

In step 906, the model receives the emission from the user from thepairing in step 904 and updates the training data to include the newinstance.

FIG. 10 is a high level block diagram showing an example of a processingdevice 1000 that can represent a system to run any of themethods/algorithms described above. A system may include two or moreprocessing devices such as represented in FIG. 6, which may be coupledto each other via a network or multiple networks. A network can bereferred to as a communication network.

In the illustrated embodiment, the processing device 1000 includes oneor more processors 1010, memory 1011, a communication device 1012, andone or more input/output (I/O) devices 1013, all coupled to each otherthrough an interconnect 1014. The interconnect 1014 may be, or include,one or more conductive traces, buses, point-to-point connections,controllers, scanners, adapters and/or other conventional connectiondevices. Each processor 1010 may be, or include for example, one or moregeneral-purpose programmable microprocessors or microprocessor cores,microcontrollers, application specific integrated circuits (ASICs),programmable gate arrays or the like, or a combination of such devices.The processor(s) 1010 control(s) the overall operation of the processingdevice 1000. Memory 1011 may be, or include, one or more physicalstorage devices which may be in the form of random access memory (RAM),read-only memory (ROM) (which may be erasable and programmable), flashmemory, miniature hard disk drive or other suitable type of storagedevice, or a combination of such devices. Memory 811 may store data andinstructions that configure the processor(s) 1010 to execute operationsin accordance with the techniques described above. The communicationdevice 1012 may be or include, for example, an Ethernet adapter, cablemodem, Wi-Fi adapter, cellular transceiver, Bluetooth transceiver or thelike, or a combination thereof. Depending on the specific nature andpurpose of the processing device 1000, the I/O devices 1013 can includedevices such as a display (which may be a touch screen display), audiospeaker, keyboard, mouse or another pointing device, microphone, camera,etc.

Unless contrary to physical possibility, it is envisioned that (i) themethods/steps described above may be performed in any sequence and/or inany combination, and (ii) the components of respective embodiments maybe combined in any manner.

The techniques introduced above can be implemented by programmablecircuitry programmed/configured by software and/or firmware, or entirelyby special-purpose circuitry, or by a combination of such forms. Suchspecial-purpose circuitry (if any) can be in the form of, for example,one or more application-specific integrated circuits (ASICs),programmable logic devices (PLDs), field-programmable gate arrays(FPGAs), etc.

Software or firmware to implement the techniques introduced here may bestored on a machine-readable storage medium and may be executed by oneor more general-purpose or special-purpose programmable microprocessors.A “machine-readable medium,” as the term is used herein, includes anymechanism that can store information in a form accessible by a machine(a machine may be, for example, a computer, network device, cellularphone, personal digital assistant (PDA), manufacturing tool, any devicewith one or more processors, etc.). For example, a machine-accessiblemedium includes recordable/non-recordable media (e.g., read-only memory(ROM); random access memory (RAM); magnetic disk storage media; opticalstorage media; flash memory devices; etc.), etc.

Physical and functional components (e.g., devices, engines, modules, anddata repositories, etc.) associated with processing device 1000 can beimplemented as circuitry, firmware, software, other executableinstructions, or any combination thereof. For example, the functionalcomponents can be implemented in the form of special-purpose circuitry,in the form of one or more appropriately programmed processors, a singleboard chip, a field programmable gate array, a general-purpose computingdevice configured by executable instructions, a virtual machineconfigured by executable instructions, a cloud computing environmentconfigured by executable instructions, or any combination thereof. Forexample, the functional components described can be implemented asinstructions on a tangible storage memory capable of being executed by aprocessor or other integrated circuit chip (e.g., software, softwarelibraries, application program interfaces, etc.). The tangible storagememory can be computer readable data storage. The tangible storagememory may be volatile or non-volatile memory. In some embodiments, thevolatile memory may be considered “non-transitory” in the sense that itis not a transitory signal. Memory space and storages described in thefigures can be implemented with the tangible storage memory as well,including volatile or non-volatile memory.

Note that any and all of the embodiments described above can be combinedwith each other, except to the extent that it may be stated otherwiseabove or to the extent that any such embodiments might be mutuallyexclusive in function and/or structure. Although the present inventionhas been described with reference to specific exemplary embodiments, itwill be recognized that the invention is not limited to the embodimentsdescribed but can be practiced with modification and alteration withinthe spirit and scope of the appended claims. Accordingly, thespecification and drawings are to be regarded in an illustrative senserather than a restrictive sense.

1. A method of executing a query engagement engine comprising: generating a plurality of queries during a contest, each query based on a real-time status of the contest and having multiple associated engagement choices from which a user is enabled to make a selection; automatically pairing each said query with editorial commentary that is tailored to influence the user's selection from the multiple associated engagement choices; and displaying each query and respective associated engagement choices and editorial commentary on a graphical user interface that is configured to receive user input corresponding to the user's selection.
 2. The method of claim 1, further comprising: generating the editorial commentary based on a history of user base query engagement that indicates that the editorial commentary correlates to a likelihood that users choose a target choice of the multiple associated engagement choices.
 3. The method of claim 2, further comprising: indicating, by a machine learning model, that there is a correlation between the editorial commentary and user selection of the target choice based on the history of user base query engagement.
 4. The method of claim 3, wherein the history of user base query engagement includes user archetypes and the correlation between the editorial commentary and user selection of the target choice is performed on a user-to-user basis and based on a first archetype that a subject user belongs to.
 5. The method of claim 1, wherein the editorial commentary includes statistics relating to an entity referred to in a respective paired query.
 6. The method of claim 5, wherein the multiple associated engagement choices of a first query of the plurality of queries are a binary choice regarding whether a predicted event will occur in the contest and the statistics are framed to indicate a positive likelihood of a target choice of the binary choice occurring.
 7. The method of claim 1, wherein the editorial commentary is an isolated graphic.
 8. The method of claim 1, wherein the editorial commentary includes non-contest information relating to an entity referred to in the displayed query.
 9. The method of claim 2, further comprising: determining the target choice of a first query of the plurality of queries for the first user based on how other users have engaged with the first query.
 10. The method of claim 1, wherein the editorial commentary corresponds to the real-time status of the contest.
 11. The method of claim 2, further comprising: receiving the selection on the multiple associated engagement choices of a first query of the plurality of queries from a first user; and updating an engagement profile of the first user based on whether the selection from the first user matches the target choice, wherein said pairing for the first user is further based on the engagement profile of the first user.
 12. The method of claim 3, further comprising: training the machine learning model with training data, wherein the training data includes the history of userbase query engagement, each instance of training data includes: a pairing of a past query and a past editorial comment; and whether an active user had engaged with the target choice or not.
 13. A method of executing a query engagement engine comprising: generating a plurality of queries during a contest, each query based on a real-time status of the contest and having multiple associated engagement choices from which a user is enabled to make a selection, the multiple associated engagement choices corresponding to predictive results of the athletic contest connected to a first entity participating in the contest; generating the editorial commentary based on a history of user base query engagement that indicates that the editorial commentary correlates to a likelihood that users choose a target choice of the multiple associated engagement choices, wherein the editorial commentary includes statistics connected to the first entity automatically pairing each said query with editorial commentary that is tailored to influence the user's selection from the multiple associated engagement choices toward the target selection; and displaying each query and respective associated engagement choices and editorial commentary on a graphical user interface that is configured to receive user input corresponding to the user's selection.
 14. The method of claim 13, wherein the multiple associated engagement choices of a first query of the plurality of queries are a binary choice regarding whether a predicted event will occur in the contest and the statistics are framed to indicate a positive likelihood of a target choice of the binary choice occurring.
 15. The method of claim 13, further comprising: determining the target choice of a first query of the plurality of queries for the first user is generated based on how other users have engaged with the first query.
 16. The method of claim 13, further comprising: determining the target choice of a first query of the plurality of queries based on a projected odds of the target choice occurring as compared to other choices of the multiple associated engagement choices.
 17. The method of claim 13, further comprising: receiving the selection on the multiple associated engagement choices of a first query of the plurality of queries from a first user; and updating an engagement profile of the first user based on whether the selection from the first user matches the target choice, wherein said pairing for the first user is further based on the engagement profile of the first user.
 18. The method of claim 13, wherein said pairing is further based on a machine learning model, and the method further comprising: indicating, by a machine learning model, that there is a correlation between the editorial commentary and user selection of the target choice based on the history of user base query engagement; and training the machine learning model with training data, wherein the training data includes the history of userbase query engagement, each instance of training data includes: a pairing of a past query and a past editorial comment; and whether an active user had engaged with the target choice or not.
 19. A system of executing a query engagement engine comprising: a network interface configured to communicate with an athletic contest descriptive feed and identify a real-time status of a contest; a backend application server including a first memory, the memory including instructions that when executed cause the backend application server to generate a plurality of queries during the contest, wherein each query of the plurality of queries is based on the real-time status of the contest and having multiple associated engagement choices from which a user is enabled to make a selection, the backend application further configured to automatically pair each said query with editorial commentary that is tailored to influence the user's selection from the multiple associated engagement choices; and a client application configured to operate on mobile devices and correspond with the backend application server via the network interface, the client application further configured to display each query and respective associated engagement choices and editorial commentary on a graphical user interface that is configured to receive user input corresponding to the user's selection.
 20. The system of claim 19, wherein the memory further includes instructions that when executed cause the backend application server to generate the editorial commentary based on a history of user base query engagement that indicates that the editorial commentary correlates to a likelihood that users choose a target choice of the multiple associated engagement choices.
 21. The system of claim 19, wherein the memory further includes instructions that when executed cause the backend application server to: receive the selection on the multiple associated engagement choices of a first query of the plurality of queries from a first user; and update an engagement profile of the first user based on whether the selection from the first user matches the target choice, wherein said pairing for the first user is further based on the engagement profile of the first user.
 22. The system of claim 20, further comprising: a machine learning model that indicates that there is a correlation between the editorial commentary and user selection of the target choice based on the history of user base query engagement and informs the automatic pairing by the backend application server, the machine learning model including training data, wherein the training data includes the history of userbase query engagement, each instance of training data includes: a pairing of a past query and a past editorial comment; and whether an active user had engaged with the target choice or not.
 23. The system of claim 19, wherein the editorial commentary includes statistics relating to an entity referred to in a respective paired query.
 24. The system of claim 19, wherein the target choice of the multiple associated engagement choices of a first query of the plurality of queries is determined for a first user based on how other users have engaged with the first query. 